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Abstract 

Several  military  and  commercial  platforms  are  currently  installing  GPS  and  inertial 
navigation  sensors  concurrently  with  the  introduction  of  high  quality  visual  capabilities 
and  digital  mapping/imagery  databases.  This  enables  autonomous  georegistration  of 
sensor  imagery  using  GPS/inertial  position  and  attitude  data.  This  georegistration 
capability  enables  data  from  digital  mapping  products  to  be  automatically  overlayed  on  the 
sensor  imagery.  In  this  paper,  we  describe  the  system  architecture  for  a 
Navigation/Electro- Optic  Sensor  Integration  Technology  (NEOSIT)  software  application. 
The  design  is  highly  modular  and  based  on  COTS  tools  to  facilitate  integration  with 
sensors,  navigation  and  digital  data  sources  already  installed  on  different  host  platforms. 
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1.  INTRODUCTION 


In  this  paper,  we  describe  a  baseline  system  architecture  for  a  Navigation/Electro- Optic 
Sensor  Integration  Technology  (NEOSIT)  software  application.  This  application  is  being 
designed  to  optimally  integrate  navigation  data,  sensor  imagery  and  image  or  terrain 
database  to  estimate  and  correct  for  errors  in  each  data  source.  The  design  is  highly 
modular  and  based  on  commercial  off-the-shelf  tools  to  facilitate  integration  with  a  variety 
of  different  navigation  and  electro- optic  sensors  and  with  different  sources  of  digital 
mapping  and  imagery  databases.  The  modular  design  is  to  allow  the  NEOSIT  application 
to  be  used  with  sensors  and  navigation  already  installed  on  different  host  platforms  and 
with  digital  mapping  and  imagery  data  sources  with  varying  degrees  of  precision. 

Applications  for  this  system  exist  for  airborne  video  surveillance  where  fusion  of  the 
navigation,  electro- optic  and  digital  mapping  data  is  required  to  increase  the  accuracy  and 
timeliness  of  target  coordinates.  The  driving  requirement  for  airborne  video  surveillance 
systems  today  is  to  increase  the  accuracy  of  target  coordinates  extracted  from  the  video 
data  and  to  decrease  the  time  required  for  processing  imagery.  To  process  target 
coordinates  today,  from  systems  such  as  the  Apache  Helicopter  or  Predator  UAV  (see 
Figure  1  and  Figure  2),  the  video  data  must  be  georegistered  using  features  selected  from 
within  the  video  images  which  are  at  known  coordinates,  defined  by  NIMA’s  Digital 
Precision  Point  Database  (DPPD)  or  another  equivalent  source.  This  is  currently  time 
intensive,  requires  significant  manual  operation  and  is  also  limited  to  operation  on  images 
where  sufficient  features  are  in  view  for  georegistration  to  occur.  With  the  NEOSIT 
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application,  the  manual  intervention  is  significantly  simplified  through  the  data  fusion  and 
automated  processing  techniques  employed. 

2.  NEOSIT  SYSTEM  COMPONENTS 

The  NEOSIT  application  is  designed  to  interface  with  the  following  components,  as 
illustrated  in  Figure  3. 

GPS/Inertial  Navigation  System 

The  NEOSTT  uses  integrated  GPS/inertial  navigation  data  from  the  host  platform  to 
provide  the  geospatial  reference  data.  Integrated  GPS/inertial  navigation  systems  are  the 
core  reference  for  most  military  aircraft  today.  More  recently,  miniaturized,  man-portable 
systems  are  also  being  developed  that  integrate  GPS  with  micro- electro- mechanical 
(MEM)  inertial  instrument  technology  to  provide  the  same  integrated  navigation 
capability.  Such  systems  are  being  developed  by  the  Navy  for  use  in  precision  targeting 
(see  Figure  4)  and  are  being  considered  by  the  Army  for  applications  such  as  the  Land 
Warrior  program. 

Electro- Optic  Sensor 

The  NEOSIT  application  can  be  used  to  process  data  from  a  variety  of  different  sensors 
including  optical,  infra-red  (IR)  or  hyperspectral  devices.  These  sensors  must  only  be 
capable  of  providing  digital  data  in  a  standard  image  format  to  the  NEOSTT  application. 

The  NEOSTT  software  integrates  this  information  with  the  navigation  and  GIS  geospatial 
data  to  precisely  georegister  the  sensor  data  against  the  GPS  reference  frame.  Our  baseline 
design  is  integrated  with  a  commercial  video  camera  to  provide  a  cost-effective  COTS 
solution. 
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GIS  Geospatial  Data 

The  NEOSIT  application  can  accept  data  from  a  variety  of  different  digital  data  sources 
including  government  and  commercial.  This  includes  rectified  imagery  (such  as  the 
Controlled  Image  Base  (CIB),  Digital  Precision  Point  Data  Base  (DPPDB),  digital  terrain 
elevation  data  (DTED)  and  vector  maps  (such  as  VMAP  or  commercial  equivalents).  We 
achieve  this  by  leveraging  a  COTS  Geographic  Information  System  (GIS)  application  to 
support  the  manipulation  of  the  various  types  of  geospatial  data  to  be  fused  including 
maps,  terrain  models  and  imagery.  Our  baseline  design  includes  ESRI’s  ArcView  product 
which  supports  analysis  and  integration  of  a  wide  variety  of  commercial  and  military 
geographic  information  sources. 

The  NEOSIT  design  leverages  a  variety  of  off-the-shelf  software  products  to  provide  a 
seamless  integrated  solution  and  to  facilitate  integrating  with  a  variety  of  navigation  and 
electro-optic  sensor  types.  The  NEOSIT  application  is  designed  to  integrate  the  GPS/ 
inertial,  sensor  imagery  and  GIS  data  to  estimate  errors  in  the  inertial  solution,  calibrate  for 
sensor  errors  and  estimate  target  locations  from  the  sensor  imagery.  The  NEOSIT  solution 
also  includes  the  capability  to  calibrate  for  the  offsets  between  different  image  and  terrain 
geospatially  referenced  databases  (Table  1).  To  achieve  this,  the  architecture  leverages 
COTS  geographic  data  processing,  image  processing  and  image  understanding  software 
tools  to  develop  a  real-time  system  for  correlation  of  on-board  platform  visual  and 
navigation  systems  with  imagery  and  terrain  databases. 

3.  PRINCIPLE  OF  OPERATION 

Using  the  NEOSIT  approach,  the  georegistration  of  video  imagery  can  be  significantly 
streamlined  by  using  GPS/inertial  navigation  information  to  estimate  the  location  and 
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attitude  of  the  video  camera.  The  principle  of  operation  of  the  NEOSIT  georegistration 


algorithm  is  illustrated  in  Figure  5 . 


The  estimated  line- of- sight  to  any  feature  in  the  video  image,  derived  in  the  navigation 
(North,  East,  Down)  frame,  can  be  computed  by  transforming  the  pixel  derived  line-of- 
sight  vector  in  camera  axes  to  the  navigation  frame  using  the  inertial  attitude  data. 


Equation  1 

[ia=[P,  Py  fVjpl  +P2y+f2 

where  px  and  py  are  the  target  pixel  coordinates  derived  from  the  image  data,  and  f  is  the 
focal  length  of  the  camera  (in  pixel  units).  The  alignment  between  the  camera  frame  and 

D 

the  inertial  body  frame  is  fixed  and  is  defined  by  the  matrix  Cc  .  The  direction  cosine 
matrix  derived  from  the  inertial  data  to  transform  from  body  to  navigation  frame 
coordinates  (CbN)  can  be  used  to  compute  the  line- of- sight  from  the  camera  location  to  the 
target  location  in  navigation  frame  coordinates. 


Equation  2 


N 

B 


f  ’  =  C 


Since  the  camera  location  is  known  (xc),  the  target  coordinates  can  be  calculated  through  a 
least  squares  solution  from  multiple  image  data.  The  observed  line -of- sight  to  the  target 
provides  a  measure  of  the  offset  between  the  estimated  target  location  and  the  observed 
target  location  through  the  following  equation. 

Equation  3 

xT  m=xc(N)  +  Rlm  fl=|vr-xc| 
z  =  xT  —xT  =  xT  -xc(N)  —  R  liN) 
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This  residual  provides  a  measure  of  the  following  error  sources  which  can  be  corrected  in 
the  NEOSIT  integration  algorithm  to  allow  fusion  of  the  sensor,  navigation  and  geographic 
data. 

•  Error  in  the  initial  estimate  of  the  feature  coordinates  ( xT )  -  this  corresponds  to  errors 
in  the  GIS  data  source 

•  Error  in  the  estimate  of  the  camera  location  ( xc  )  -  this  corresponds  to  errors  in  the 
navigation  solution 

•  Error  in  the  estimate  of  the  camera  attitude  (CbN)  -  this  corresponds  to  errors  in  the 
inertial  alignment  and  misalignment  angle  errors  between  the  inertial  system  and  the 
video  camera 

4.  GIS  DATA  MANAGEMENT 

The  GIS  data  management  function  in  the  NEOSIT  application  is  performed  using  the 
ArcView®  software  product  provided  by  ESRI.  This  allows  both  raster  and  vector  GIS 
data  sources  to  be  read  in  a  variety  of  different  formats  including  those  listed  in  Table  2. 
The  following  GIS  functions  are  performed  by  the  NEOSIT  application. 

GIS  Data  Management 

This  function  manages  the  storage  and  retrieval  of  the  GIS  data  relevant  to  the  particular 
mission.  This  is  coordinated  with  the  current  region  of  interest  using  the  navigation 
coordinates  provided  from  the  NEOSIT  navigation  sensor.  Retrieval  and  management  of 
the  different  data  sources  is  accomplished  using  the  Avenue™  programming  language. 
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GIS  Feature  Selection 


The  GIS  feature  selection  algorithm  operates  using  a  GIS  query  to  select  the  identified 
types  of  features  from  the  GIS  database  that  fall  within  the  field  of  view  of  each  image. 

This  information  is  used  by  the  image  processing  function  to  narrow  the  search  focus  and 
analysis  to  be  performed  and  to  optimize  the  algorithm  applied  for  the  type  of  feature  being 
identified.  These  features  can  include  easily  identifiable  objects  such  as  signage,  road 
intersections,  bridges,  or  background  features  such  as  mountains  or  buildings.  Using  the 
DLL  interface,  the  GIS  feature  selection  software  can  direct  ArcView®  to  load  and  display 
selected  features  or  regions  of  interest  from  the  image  and  terrain  database  files.  In  Figure 
6,  data  is  shown  loaded  from  multiple  Digital  Ortho  Quadrant  files. 

GIS  Data  Visualization 

To  facilitate  correspondence  matching  between  the  sensor  data  and  the  digital  mapping 
data,  the  NEOSIT  application  uses  the  navigation  sensor  position  and  attitude  information 
to  map  the  2D  display  into  a  3D  perspective  view  image.  The  corresponding  3D  view  for  a 
typical  sensor  location  is  shown  in  Figure  7  mapped  from  the  2D  data  shown  in  Figure  6. 

5.  IMAGE  PROCESSING 

The  core  function  to  be  performed  by  the  image  processing  software  is  to  observe  the  pixel 
coordinate  offsets  between  the  predicted  location  of  a  feature  in  the  image  and  the  actual 
location  of  the  same  feature.  This  observation  enables  either  the  navigation  errors  to  be 
identified  from  known  feature  locations  or,  in  the  reverse  application,  to  use  the  pixel 
offset  to  identify  errors  in  the  recorded  location  for  that  feature. 
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The  basic  steps  to  be  followed  for  generating  this  observation  are  illustrated  in  Figure  8. 


First  a  Model  is  generated  of  the  feature  to  be  matched.  Using  the  a-priori  information  on 
the  sensor  field  of  view  the  correct  aspect  angle  and  pose  of  this  model  is  created.  The 
predicted  locations  of  the  feature  to  be  matched  are  also  used  to  Extract  the  region  of 
interest  from  the  sensor  imagery  to  aid  the  processing  function.  Finally  the  extracted  data 
is  matched  to  the  model  to  observe  the  offset  between  the  two  data  sources. 

The  symbolic  processing  methodology  applies  equally  well  to  processing  and  aligning 
digital  mapping  data  from  both  vector  and  raster  imagery  sources,  as  illustrated  in  the 
following  examples. 

Feature  Matching  using  Vector  Data 

The  first  example  illustrates  how  actual  sensor  imagery  can  be  automatically  aligned  with  a 
vector  data  model.  The  first  step  in  the  process  is  to  generate  the  model  data.  The 
ArcView  program  was  used  to  generate  a  view  of  the  road  data  over  the  same  area  covered 
by  the  DOQ  view.  This  reference  model  is  shown  in  Figure  9  generated  from  TIGER  96 
data  files.  A  region  of  interest  (ROI)  is  selected  to  identify  the  road  segment  to  be  used  for 
the  matching  process.  The  ROI  is  shown  as  the  blue  area  in  Figure  9  and  is  converted  by 
the  NEOSIT  application  to  a  template  for  cross  correlation  with  the  sensor  image.  For  this 
example  overhead  sensor  imagery  was  used  to  demonstrate  the  alignment  process  between 
the  two  data  sources. 

In  Figure  10  the  DOQ  input  image  is  shown  with  a  ROI  selected  that  includes  the  template 
segment.  This  ROI  is  a  larger  area  to  account  for  possible  data  errors  between  the 
reference  model  data  source  and  the  sensor  imagery.  The  image  data  is  digitally  enhanced 


to  highlight  the  road  segments  which  will  be  used  for  cross- correlation.  The  filtered  image 
data  is  shown  in  Figure  1 1 . 

The  NEOSIT  matching  function  is  used  to  search  for  the  template  ROI  over  the  image 
ROI.  In  Figure  12  the  location  of  the  template  match  with  the  image  data  is  shown.  In  this 
case,  the  image  processing  showed  that  there  was  a  4  pixel  or  roughly  12  meter  offset 
between  the  two  data  sets  at  these  coordinates. 

This  example  has  shown  how  the  NEOSIT  matching  function  can  automatically  match 
road  segments  from  a  digital  vector  data  source  to  a  digital  sensor  image  of  the  same  area. 

A  similar  process  can  be  employed  using  raster  data  mapped  to  a  3-D  perspective  as  shown 
in  Figure  7  to  extract  the  model  used  for  template  matching  in  place  of  the  vector  data 
model  generated  in  Figure  9. 

Feature  Matching  Using  DEM  Data 

In  this  example,  digital  elevation  model  (DEM)  data  provided  by  the  USGS  for  the  Palmer 
Fake  area  was  used  to  generate  a  3-D  perspective  view  from  the  location  of  an  image 
provided  by  NAVSYS’  GI-Eye  GPS/inertial/video  system  (Brown,  Zhang  and  Reynolds, 
1999). 

The  first  step  in  this  process  is  to  extract  a  skyline  profile  from  the  DEM  data  to  be  used  as 
the  reference  model  for  template  matching  on  the  GI-Eye  image.  This  is  mapped  to  the 
same  perspective  as  the  image  using  the  GI-Eye  position  and  attitude  data.  The  reference 
model  generated  from  this  data  is  shown  in  Figure  13  overlay ed  on  the  GI-Eye  image  data. 
The  next  step  in  this  process  is  to  extract  a  symbolic  representation  of  the  image  to  be  used 
in  the  NEOSIT  matching  process.  The  results  of  this  step  are  plotted  on  Figure  13  in 


9 


yellow.  As  can  be  seen,  there  is  a  significant  offset  in  the  horizontal  direction  between  the 
model  and  the  image. 

The  NEOSIT  matching  algorithm  was  used  to  observe  the  pixel  offset  between  the  model 
data  and  the  image.  This  algorithm  observed  an  offset  of  (48,0)  pixels  between  the  two 
data  sources.  In  Figure  14  the  Model  data  corrected  for  this  offset  is  shown  overlaid  on  the 
segmented  image  data. 

6.  NEOSIT  MODES 

The  NEOSIT  software  has  two  modes  of  operation  for  applying  the  pixel  offset  observed 
from  the  Image  Processing  function.  Both  use  optimal  estimation  techniques  to  observe 
system  error  components.  The  first  mode  (GIS  Mode)  is  illustrated  in  Figure  15.  In  this 
mode,  the  navigation  solution  is  used  to  provide  the  geospatial  reference  frame  and  the 
pixel  offsets  are  used  to  compute  the  location  error  of  GIS  or  “target”  features  extracted 
from  the  sensor  data.  This  data  is  used  to  build  a  database  which  can  correct  for  the  base- 
location  errors  in  the  GIS  data  source  to  improve  the  accuracy  of  the  digital  map  products. 
With  sufficient  data,  the  digital  map  errors  shown  in  Table  1  can  be  reduced  to  GPS 
accuracy  levels. 

The  second  software  mode  is  designed  to  enable  the  navigation  sensor  errors  to  be  updated, 
for  example  during  periods  where  the  GPS  signals  may  be  denied.  In  this  mode,  the 
processing  steps  are  illustrated  in  Figure  16.  The  pixel  offsets  from  features  extracted  from 
the  sensor  data  are  used  to  estimate  the  errors  in  the  onboard  navigation  solution  relative  to 
the  georegistered  coordinates  of  these  points  from  the  corrected  digital  mapping  data. 


10 


Again,  optimal  estimation  techniques  are  used  to  estimate  the  different  error  coordinates 
from  the  pixel  offset  data  output  by  the  Image  Processing  function. 

7.  CONCLUSION 

The  NEOSIT  application  is  being  developed  under  a  Small  Business  Innovative  Research 
(SBIR)  contract  award  for  CECOM,  Ft  Monmouth,  New  Jersey.  Under  the  first  phase  of 
this  program,  the  NEOSIT  system  was  prototyped  using  NAVSYS’  GPS/inertial  video 
targeting  system,  the  GI-Eye  (see  Figure  17).  Under  the  next  phase  of  this  program,  the 
NEOSIT  will  be  developed  for  use  as  a  generic  integrating  application  for  fusing 
navigation,  electro-optic  sensor  and  digital  mapping  product  data.  This  system  will  be 
designed  for  automated  operation  to  enable  real-time  correction  of  digital  map  coordinates 
and  to  provide  a  backup  navigation  solution  in  the  event  of  GPS  denial. 


it 
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TABLES 


Table  1  NIMA  Mapping  Product  Summary1 


CIB 

DTED  Level  1 

VMAP  Level  2 

Digital  PPDB 

Output  Unit 

1  degree  cell 

1  degree  cell 

15’  tile 

1  degree  ZDR 

Scale 

1:75K 

1:250K 

1:50K- 1:100K 

1:110K 

Digital  Storage 

13  GB 

2.9  MB 

varies 

26  GB 

Horiz  Accuracy 

<  61  m 

50  m 

50-200  m 

Classified 

Vert  Accuracy 

N/A 

30  m 

0.5-2  contours 

Classified 

Table  2  NEOSIT  GIS  Data  Sources 


Military  Products 

Commercial  Products 

Controlled  Image  Base  ™  (CIB  ™) 

Digital  Terrain  Elevation  Data  (DTED) 

Vector  Smart  Map  Product  (VMAP) 

Digital  Orthophoto  Quadrangles  (DOQ) 
Digital  Elevation  Model  (DEM) 

Digital  Raster  Graphics 

TIGER®  95 

1  “Mapping,  Charting  &  Geodesy  Handbook”,  Naval  Aviation  Systems  Team,  AVDEP-HDBK-12, 
Version  2  (http://www.fas.org/man/dod-101/navy/docs/pc  hndbk.html) 
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FIGURES 


Figure  1  Predator  UAV 


Figure  2  Apache  Helicopter 


Figure  3  NEOSIT  Component  Interfaces 
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Figure  4  SPOTS  Man-portable  GPS/IMU  Targeting  Sensor 


Figure  5  GPS/Inertial  Georegistration 
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Figure  6  ArcView  2D  view  window  with  5  themes  loaded 


Figure  7  ArcView  3D  view  window  looking  west  from  Larkspur  1-25  interchange 
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Figure  8  Symbolic  Processing  Methodology 


Figure  9  Reference  Model  with  feature  selected 
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Figure  10  Input  Image  with  area  to  search  selected 


Figure  11  Input  image  after  filtering  to  bring  out  roads 
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Figure  12  Input  image  with  feature  found  within  the  search  region  using  cross¬ 
correlation 
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Figure  13  GI-Eye  data  with  DEM  model  superimposed 
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Figure  14  Symbolic  image  representation  and  Matched  model 


Figure  15  GIS  Update  Software  Mode 
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Figure  16  Navigation  Update  Software  Mode 


Figure  17  GI-Eye  GPS/Inertial/Video  Targeting  System 
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